1. Field of the Invention
The present invention relates to image processing for correcting (or reducing) a deterioration component of an image generated by image pickup, utilizing image restoration processing.
2. Description of the Related Art
An image obtained by capturing an object by an image pickup apparatus, such as a digital camera, contains a blur component as an image deterioration component, such as the spherical aberration, coma, curvature of field, and astigmatism of an image pickup optical system. This blur component occurs when a light flux emitted from one point of the object, which is expected to converge on one point on an image pickup surface when there is no aberration or diffraction, forms an image with some divergence.
The blur component, as used herein, is optically represented by a point spread function (“PSF”) and is different from a blur caused by defocusing. A color shift in a color image caused by longitudinal chromatic aberration, chromatic spherical aberration, or chromatic coma of the image pickup optical system is a difference of the blurring degree among wavelengths of the light. A lateral color shift can be regarded as a position shift or phase shift due to a difference of the image pickup magnification for each wavelength of the light if the lateral color shift is caused by the lateral chromatic aberration of the image pickup apparatus.
An optical transfer function (“OTF”) obtained by performing a Fourier transform for the PSF is frequency component information of the aberration and is represented by a complex number. An absolute value of the OTF or an amplitude component is referred to as a modulation transfer function (“MTF”) and a phase component is referred to as a phase transfer function (“PTF”). The MTF and the PTF are frequency characteristics of an amplitude component and a phase component of the image deterioration due to the aberration. The phase component will be expressed as a phase angle by the following equation, where Re(OTF) and Im(OTF) are a real part and an imaginary part of the OTF:PTF=tan−1(Im(OTF)/Re(OTF))
Thus, the OTF of the image pickup optical system deteriorates the amplitude component and the phase component of the image, and the deteriorated image has an asymmetrical blur state of points of the object like the coma.
In addition, the lateral chromatic aberration (i.e., chromatic aberration of magnification) occurs when an imaging position shifts due to a difference of an imaging magnification for each wavelength of the light, and is obtained as RGB chromatic components, for example, in accordance with the spectral characteristic of the image pickup apparatus. The image divergence occurs not only when the image position shifts among the RGB, but also when the imaging position shifts for each wavelength of a chromatic component or when the phase shifts. Therefore, due to the phase degradation component of the aberration, asymmetry of the PSF is viewed on a primary section of each direction (azimuth direction) orthogonal to a principal ray (a ray that passes the center of the pupil of the image pickup optical system). The amplitude degradation component affects the size of the spread of the PSF for each azimuth direction.
In order to highly precisely correct through image processing the image degradation caused by the image pickup optical system, it is thus necessary to correct the phase degradation component and the amplitude degradation component of the aberration.
Edge enhancement processing is a known method of correcting the amplitude degradation component by detecting an edge portion in an image and by enhancing the edge as disclosed in Japanese Patent Laid-Open No. (“JP”) 2000-156816.
A geometric correction is a known method of correcting a phase degradation component by varying an image magnification for each chromatic component of an image as disclosed in JP 06-113309.
Another known method of correcting the amplitude degradation component and the phase degradation component utilizes the OTF information of the image pickup optical system. This method is referred to as an image restoration or image recovery, and processing used to correct (or reduce) an image degradation component utilizing the OTF information will be referred to as image restoration processing hereinafter.
The image restoration processing will be outlined as follows: Assume that g(x,y) is a deteriorated image (input image), f(x,y) is a non-deteriorated, original image, and h(x,y) is the PSF as a Fourier pair of the OTF. Then, the following equations are established, where * denotes a convolution and (x,y) denotes a coordinate on an image.g(x,y)=h(x,y)*f(x,y)
The above equation is converted into a display format on a two-dimensional frequency plane by a Fourier transform, which is a format of a product for each frequency. “H” denotes a Fourier-transformed PSF or corresponds to the OTF. (u,v) denotes a coordinate on the two-dimensional frequency plane or corresponds to a frequency.G(u,v)=H(u,v)·F(u,v)
In order to obtain the original image from the deteriorated image, both sides are divided by H as follows:G(u,v)/H(u,v)=F(u,v)
When this F(u,v) is inverse-Fourier-transformed and returned to the real surface, the restored image corresponding to the original image f(x,y) can be obtained.
Where R denotes an inverse Fourier transform of H−1, the original image can also be obtained through the convolution processing to the image on the real plane as in the following equation:g(x,y)*R(x,y)=f(x,y)
This R(x,y) is referred to as an image restoration filter. For a two-dimensional image, the image restoration filter is usually a two-dimensional filter having a tap (cell) corresponding to each pixel of the image. In general, as the number of taps in the image restoration filter increases, the restoration precision improves. The image restoration filter needs to reflect at least an aberrational characteristic, and is remarkably different from a conventional edge enhancement filter (high-pass filter) having about three taps in each of the horizontal and vertical directions. Since the image restoration filter is prepared based on the OTF, the deterioration of each of the amplitude component and the phase component can be highly precisely corrected.
The real image has a noise component, and the noise component is remarkably amplified disadvantageously along with the restoration of the deteriorated image when the restoration utilizes the image restoration filter made with a perfect reciprocal of the OTF. With noises, an image worth viewing cannot generally be obtained.
This can be expressed by the following equations where N denotes a noise component:G(u,v)=H(u,v)·F(u,v)+N(u,v)G(u,v)/H(u,v)=F(u,v)+N(u,v)/H(u,v)
For example, like a Wiener filter illustrated by Equation 1, there is a known method of controlling the restoration degree (or restoration level) in accordance with a signal to noise ratio (“SNR”) between an image signal and a noise signal:
                              M          ⁡                      (                          u              ,              v                        )                          =                              1                          H              ⁡                              (                                  u                  ,                  v                                )                                              ⁢                                                                                      H                  ⁡                                      (                                          u                      ,                      v                                        )                                                                              2                                                                                                              H                    ⁡                                          (                                              u                        ,                        v                                            )                                                                                        2                            +                              SNR                2                                                                        Equation        ⁢                                  ⁢        1            
M(u,v) denotes a frequency characteristic of the Wiener filter, and |H(u,v)| denotes an absolute value (MTF) of the OTF. For each frequency, this method further restrains the restoration gain as the MTF is smaller or the SN ratio is lower, and further enhances the restoration gain as the MTF is larger or the SN ratio is higher. In general, the MTF of the image pickup optical system is high on the low frequency side and low on the high frequency side, and thus the restoration gain of the image is restrained substantially on the high frequency side.
When the noises are impermissible to the desired image quality after the restoration processing is performed by setting the actual SNR to a parameter, an additional restraint on the noise amplification is necessary by adjusting the SNR parameter so as to reduce the restoration gain. This corresponds to the SNR value in Equation 1 set to a larger value. In addition, when there is a difference between the aberrational characteristic in the actual image pickup condition and the aberrational characteristic estimated by the image restoration filter, the restored image may contain a problem of an artifact, such as ringing. Moreover, when the restoration degree is different from the estimated one for each of the RGB chromatic components, the restored image may be colored or a false color may occur. This false color is also one of the artifacts, such as noises and ringing.
FIG. 15 illustrates false colors in a restored image. The false color occurs as illustrated by a broken line in FIG. 15. A false color “A” occurs when an edge part is further colored than a pre-restoration image caused by the manufacturing error. A false color “B” occurs when a periphery of a brightness saturation part is further colored than the pre-restoration image caused by the brightness saturation generated by the intensity of solar reflected light. A false color “C” occurs when the edge part is further colored than the pre-restoration image caused by defocusing.
In the Wiener filter, the restoration degree lowers by setting a large value to the SNR value in Equation 1. The restoration degree starts dropping from a high frequency and the pre-restoration frequency characteristic cannot be reproduced even when the restoration degree is reduced. This is clear from the fact that M=1 is not established or the image restoration filter does not affect the image unless the SNR has a frequency characteristic in the Equation 1.
In order to adjust the restoration degree, JP 2007-183842 sets a parameter used to design the image restoration filter as in Equation 2:
                              F          ⁡                      (                          u              ,              v                        )                          =                                                            α                ⁢                                                                  ⁢                                                      H                    ⁡                                          (                                              u                        ,                        v                                            )                                                        *                                            +              1              -              α                                                      α                ⁢                                                                                                H                      ⁡                                              (                                                  u                          ,                          v                                                )                                                                                                  2                                            +              1              -              α                                ×                      G            ⁡                          (                              u                ,                v                            )                                                          Equation        ⁢                                  ⁢        2            
F(u,v) and G(u,v) are Fourier transforms of the restored image and the deteriorated image, respectively. The adjustment parameter α provides a change from a filter that affects nothing (α=0) to an inverse filter (α=1), and the restoration degree of the image is adjustable in a range from the original image to the maximum restored image.
The image restoration method disclosed in JP 2007-183842 is a continuously adjusting method of the restoration degree including a reproduction of the original image. Apparently, this adjustment can successfully restraint generations of noises, ringing, and a false color. However, this image restoration method requires a recalculation of an image restoration filter and additional convolution processing to an input image whenever the parameter is changed.
In addition, since the deteriorated image is closer to the original image as the restoration degree is made lower, an aberrational component having asymmetrical blur again appears, such as the lateral chromatic aberration and the coma. The asymmetry of the aberration is caused by a phase shift for each frequency for forming the PSF. On the image, the coma degrades the image quality due to the deletion of an image, and the lateral chromatic aberration degrades the image quality through a chromatic shift. In particular, the lateral chromatic aberration is likely to stand out, and has conventionally been corrected as in a method disclosed in JP 06-113309. The method disclosed in JP 2007-183842 can correct the lateral chromatic aberration only when the restoration degree is high, and the lateral chromatic aberration becomes larger as the restoration degree is lower.
The adjustable restoration degree can restrain the problem, but increase the asymmetry of the aberration, such as the lateral chromatic aberration. In addition, the processing load increases because a lateral chromatic aberration amount varies and a correction amount of the lateral chromatic aberration needs to be changed in accordance with the setting of the restoration degree even when the lateral chromatic aberration correction processing is introduced in addition to the image restoration processing.